The pair, who are fully vaccinated, left shortly before Vice President Kamala Harris was set to be interviewed in-studio